Marina, Davis-Hamilton, Charmanski 95 


Evaluating Research Administration: Methods and Utility 


Sarah Marina, MA (in progress) 

Tufts University 

Zoya Davis-Hamilton, EdD, CRA 

Tufts University 

Kara E. Charmanski, MBA 

Tufts University 

Abstract: Three studies were jointly conducted by the Office of Research Administration 
and Office of Proposal Development at Tufts University to evaluate the services within each 
respective office. The studies featured assessments that used, respectively, (1) quantitative 
metrics; (2) a quantitative satisfaction survey with limited qualitative questions; and (3) 
a mixed-methods survey that evaluated both satisfaction and learning using quantitative 
metrics and equally weighted qualitative responses. These studies are used as case studies, 
and are described congruently to demonstrate that varying assessment methods have utility, 
separately and together, in evaluating the research administration enterprise. Findings 
illustrate that several factors influence which method should be used, including the goals of 
the evaluation itself. Additionally, findings indicate that judicial use of quantitative metrics 
supplemented with qualitative tneasures in mixed methods approaches allows the user to paint 
a more comprehensive and detailed picture. Finally, methods and inferencesfrom such studies 
can be leveraged to gain or sustain competitive advantage among peer institutions and position 
the research administration enterprise for future success. 

Introduction 

Metrics are “a means of representing a quantitative or qualitative measurable [emphasis in 
original] aspect of an issue in a condensed form” (Horvath, 2003, as cited in Kreimeyer & 
Lindemann, 2001, p. 75). Consequently, performance metrics represent “[m]easures used to 
evaluate and improve the efficiency and effectiveness of business process” (Cole, 2010, p. 14). 
Examples of quantitative metrics used in the field of research administration include success rate 
(number of submitted proposals accepted for funding), dollar amount of funding applied for and 
received, and number of applications submitted. Customer feedback on research administration 
services is an example of qualitative metrics. The benefits of developing and implementing 
metrics for research administration offices include defining and monitoring business processes 
and their impact, defining responsibilities, managing expectations, improving decision making 
and prioritization, motivating teams and evaluating staff performance (Haines, 2012). These 
benefits can be condensed to three areas: changing behavior, driving performance, and supporting 
investments in research administration (Taylor, Lee, & Smith, 2014, slide 5). 
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Current use of metrics in evaluating research administration 

Analyzing metrics in relation to sponsored funding and measuring research productivity is a 
well-established business practice among academic institutions with a research mission or focus. 
The University of Minnesota, for example, tracks data related to expenditures; publications and 
indicators of faculty reputations; proposals and grant awards; invitations and collaborations; 
indirect cost recovery; student engagement in research; space allocations; and other “common 
research metrics” (University of Minnesota, 2008, p. 10). Some institutions have incorporated 
metrics into their daily operations. The University of Iowa posts weekly “Homepage Metrics” 
on its Division of Sponsored Programs’ website (http://dsp.research.uiowa.edu). These metrics 
consist of the numbers of routing forms that were received; submitted proposals; completed 
contracts; non-monetary agreements and subawards; and processed awards, and are calculated 
weekly and during the fiscal year to-date. 

Those institutions that do not already use metrics to guide and evaluate their work are now 
outside the norm. A recent informal survey of research administrators for the Society of Research 
Administrators (SRA) International’s electronic newsletter, Catalyst, found that most research 
administration offices (78% of those who responded) conduct some kind of evaluation of their 
services (Davis-Hamilton, 2014). The most commonly used evaluation methods reported include 
collection of informal feedback from customers, examination of existing management reports 
and data, and comparison of current internal operational data to those from prior periods. 

Pitfalls of current metrics used to evaluate research administration 

While the metrics discussed above can be useful and informative assessment tools, some scholars 
feel that metrics based on financial or other quantitative measures “do not sufficiently capture 
the quality of the level of service demands” placed on research administration (Cole, 2010, p. 
viii). By “reducing the complexity of the representation of an issue” quantitative metrics “tend 
to oversimplify or omit dependencies of an issue, thus making the representation incomplete” 
(Kaplan & Norton, 1992, as cited in Kreimeyer & Lindemann, 2001, p. 87). 

Furthermore, the external environment influences traditional quantitative metrics, like success 
rates, making it difficult to evaluate the merit of the activities internal to the institution. This can 
be illustrated by looking at success rates from the perspective of the PESTEL framework, a tool 
used to identify the external opportunities and threats that may impact an institution’s operation. 
The PESTEL framework organizes these external “forces” into six major categories: Political, 
Economic, Socio-cultural, Technological, Ecological, and Legal (Rothaermel, 2013, pp. 56-57). 
These forces can drive funds available to support research, which then influences success rates in 
the public and private sectors. 

Research programs outside of a sponsoring agency’s priority areas face increased challenges in 
securing funds. An example is the recent emphasis placed on obesity research by the United States 
Department of Agriculture and the National Institutes of Health, linked to the obesity epidemic 
in the US and the current administration’s personal interest in tackling this public health challenge 
(socio-cultural and political forces). This presents an opportunity and competitive advantage for 
organizations with active obesity research programs, increasing their success rate. Conversely, it 
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serves as a threat, and competitive disadvantage, to other health sciences organizations, lowering 
their success rate. Similarly, the more recent decline in economic growth has culminated in 
sequestration and fewer research dollars, lowering success rates nation-wide and threatening the 
breadth and longevity of many research programs (an economic force). The influence of these 
external “forces” must, therefore, also be considered when using quantitative metrics to evaluate 
an institutions research administration enterprise. While many research administration offices 
currently use metrics in evaluating their work, the need for an effective, evidence-based metric 
standard that captures the complexity of the field remains unmet. Adoption of a mixed methods 
approach, utilizing both quantitative and qualitative measures, may allow research administrators 
to garner more comprehensive evaluations of their services, either individually or collectively as 
representative of the research administration enterprise. 

The search for effective metrics to evaluate research administration: complexity metrics 
& satisfaction 

The current lack of standard performance metrics for research administration services has far- 
reaching consequences. According to a recent SRA Catalyst survey (Davis-Hamilton, 2014), 15% 
of those who conduct evaluations of their offices have doubts about their validity. As noted in the 
same Catalyst article, some common platforms for evaluation were utilized but no clear standards 
emerged from the results of their informal survey. This lack of standard metrics not only creates 
validity concerns, but also makes comparisons across offices more difficult. 

The ability to assess the quality of a research administration enterprise is extremely important. 
It is critical to ensure that available research administration resources adequately support 
investigators. As well, such assessment can inform decisions on allocation of additional resources 
that meet the changing needs of faculty and drive competitive advantage. To quote Janice Besch, 
Managing Director of the National Institute of Complementary Medicine at the University of 
Western Sydney, “[r]esearchers require robust management systems to support their activities in 
a funding environment that is highly competitive and carrying a significant compliance burden. If 
they are not well supported, they are likely to scale down, or fail in, their grant seeking activities; 
funding will diminish; and there is a risk that whole research programs could be shut down due 
to compliance breaches” (Besch, 2014, p. 1). Management systems to support research activities 
can include the various software and other technological tools research administrators use, 
but more fundamentally can be viewed as the research administrators themselves and research 
administration as a whole. Proper assessment of the quality of such instrumental systems, as part 
of a comprehensive effort to optimize them, is part of the effort to diminish those threats to 
which research activities are vulnerable, according to Beschs observations. Another study (Cole, 
2010) further posits that the success of research administration offices should be measured 
by performance metrics grounded in needs and preferences of both faculty and department 
administrators. Below, we assess two solutions to meet this requirement: complexity metrics and 
satisfaction surveys. 

Complexity metrics. In the quest to develop meaningful metrics of research administration, one 
must take into consideration the complexities of the tasks performed by research administrators. 
In this context, the judgment of complexity is in relation to the “more complex grant awards, 
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which require more time and resources to manage due to the nature of sponsor requirements 
and/or collaborative activities with multiple researchers and institutions” (Taylor, Lee & Lee, 
2014, slide 30). One means of measuring complexity was compiled by Chris Thomson of 
Moderas (moderas.org). His Proposal Complexity Scoring Matrix aims to judge the complexity 
of a proposal by the workload that goes into its review, which includes factors related to staffing, 
budget, human or animal subjects, subcontracts, international collaborations, and others. 

Duke University offers another approach to measuring complexity, where management measures 
the complexity of a department’s sponsored research portfolio and ties it to the compensation 
of research administrators. The information on complexity and types of grants is supplemented 
“with information about the department’s practices with training, hiring, and procurement. Each 
assessment received a score, and the overall score is averaged” (Melin-Rogovin, 2012). A lower 
score is better, as it indicates that the department manages its portfolio well, has appropriate skills 
and training, is hiring adequately, and uses existing systems to maintain compliance. 

Despite painting a more comprehensive picture of the research administration enterprise, 
complexity metrics take significant time and effort to develop and may be cumbersome and 
time-consuming for end-users. Additionally, it is highly subjective how much weight should be 
assigned to various components of such metrics in evaluations of research administration. Thus, 
while complexity metrics are the most nuanced quantitative metrics and often present a more 
accurate account of productivity and performance, they alone may not always be sufficient for a 
comprehensive assessment. 

Satisfaction surveys. In a departure from quantitative and objective complexity metrics, 
satisfaction metrics focus on how satisfied faculty and staff are with the services provided to 
them by offices of research administration and development. This information can be assessed 
using both quantitative and qualitative measures. Giese and Cote (2002) define the concept of 
customer satisfaction as “a summary affective response of varying intensity” with a “time-specific 
point of determination and limited duration, that is directed toward focal aspects of product- 
related experience” (p. ii). hi the research administration context, customers are faculty and their 
support staff that use services of research administration offices. Along these lines, product-related 
experience is the research administration service itself and the value provided to the customer 
through the use of said service. 

There exist few scholarly studies on faculty’s own perceptions of the administrative resources 
available to them in the conduct of research (Mullen, Murthy, & Teague, 2008). This leaves a large 
gap in any attempt to assess the success of research administration offices. As Dr. Sharon Cole 
notes in the context of describing changes in the research administration system to bring about 
growth and collaboration, “[rjesearch faculty are the generators of the grants administration 
workload and recipient of services; therefore, their opinions and participation are deemed 
important to the improvements of the system of research administration” (Cole, 2007, p. 19). 
Besch (2014) continues this line of argument, writing, “[a]n effective barometer for excellent 
research management will undoubtedly be how it is perceived by those who rely on it” (p. 1). 
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Despite this dearth of formalized scholarship, satisfaction surveys—surveys that include a faculty 
opinion component—have been used at a variety of institutions to evaluate institutional research 
resources. These include Coles study of faculty at several research universities, which used a non- 
statistically significant informal online survey distributed via email asking faculty what services 
they found lacking in existing research administration structures, and what changes were needed 
in the working relationship between administrators and investigators (Cole, 2007). Besch (2014) 
moved beyond a customer satisfaction survey to introduce a short, clearly written mixed methods 
study that she reports, “never failed to produce a rich set of responses and a real sense of how the 
office was performing” (p.4). Overall those who use qualitative measures asking for client feedback 
about and satisfaction with research administration services report high-quality, useful data. 

Limits of this largely qualitative and subjective method include a relatively low return rate for 
surveys among faculty. One survey reported a 20.6% response rate (Mullen et. al., 2008), another 
only 11% (Cole, 2007), while a third survey of both faculty and administrators rose from 20% 
to 35% over several administrations (Besch, 2014). However, as qualitative data involves rich and 
detailed responses, high quality and analytically valuable information can be gathered from a 
small response group despite the potential lack of statistical power. 

Methods and Results 

We present below three case studies, the first and second conducted by the Office of Research 
Administration, and the third conducted by the Office of Proposal Development at Tufts 
University. Each presents the methods and results of a unique metric analysis and/or a customer 
satisfaction survey designed to assess key metrics to analyze the success of services within each 
respective office. While each of these studies was a standalone effort with its own objectives, our 
goal in describing them together is to analyze the utility of varying methods in evaluating research 
administration of (l) quantitative metrics; (2) a quantitative satisfaction survey with limited 
qualitative questions; and (3) a mixed-method survey that evaluated satisfaction and learning 
using quantitative questions and equally weighted qualitative responses. In our discussion, we 
will assess this utility, as well as the limitations of each method, followed by examples of how 
inferences drawn from each can be used to gain or sustain competitive advantage. 

Case Study 1: Office of Research Administration (ORA) Metrics Study 

Background. The Office of Research Administration (ORA) provides pre-award and non- 
financial post-award services to Tufts University faculty and department administrators. ORA is 
representative of two distinct offices: one that serves the Boston Campus (Health Sciences) and 
the Grafton Campus (Veterinary Medicine), and one that serves the Medford Campus (Arts & 
Sciences, Engineering, The Fletcher School of Law and Diplomacy, and several Centers). 

Methods. In order to assess the performance of the office on a variety of factors, a metrics study 
took place on the Tufts University Boston Campus during a three-month period of the 2013 
calendar year (June through August), representing a quarter-cycle of annual pre-award activity. 
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The study tracked metrics relating to the four core ORA services: (l) review and submission of 
proposals; (2) processing of requests to change the awards budget (referred to as “re-budgets” 
below); (3) processing of requests to extend the award end date without additional costs (referred 
to as “no-cost extension requests” below); and (4) issuing sub-award agreements to other 
institutions where Tufts is the prime entity. 

In order to quantitatively assess the delivery of these services, each of the four ORA-Boston staff 
members recorded twice a day all their actions relevant to each of the designated activities (see 
Table 1). Dates of actions were recorded in an Excel spreadsheet located on a shared drive, and 
each staff member was scheduled for unhindered daily access during two one-hour intervals, one 
in the morning and one in the afternoon. Once recorded, the data was used to map and analyze 
the processes as they progressed over time. It was hypothesized that analysis of metrics relating 
to documentation flow and length of time for each action would allow for the identification of 
bottlenecks and opportunities for streamlining as well as improving service delivery. 

Results. Results of the study supported the hypothesis, as the metrics tracked uncovered 
addressable points of delay in each of the areas examined. For example, it was found that 33% 
of proposals during the study period arrived to ORA for review on the day of the submission 
deadline. This prevented the full review of these proposals and was correlated with an average 
resubmission rate of 1.27 (in this example, a resubmitted proposal is one which must be sent to 
a given funding agency more than once due to errors, warnings, or other deficiencies discovered 
shortly after the initial submission). It was concluded that additional time for review could reduce 
or eliminate the resubmission rate. 

Results also indicated that 35% of re-budget requests made during the study period required 
revisions or additional information from the department before they were ready for processing by 
ORA. Further analysis revealed that only 11.5% of re-budgets processed during the study period 
required ORA involvement, as agency approval was unnecessary in the majority of the cases. 
Similarly, results captured that nearly half (44%) of no cost extension requests required additional 
information that had to be subsequently collected, increasing the time needed to complete the 
process. Finally, metrics concerning the sub-award process revealed that the largest bottlenecks 
were centered on the processes of gathering necessary information from a department and waiting 
for countersignature from the other party. 

Case Study 2: ORA Customer Satisfaction Survey 

Background. See background for Case Study 1. 

Methods. In a separate effort following the case study described above, both Boston/Grafton and 
Medford ORA locations distributed an IRB-approved satisfaction survey during the three weeks 
in September-October 2013 to all three of Tufts’ campuses. As ORA is a central pre-award office, 
customers for the purposes of this study were defined as: (l) faculty that either applied for or 
received at least one grant or other extramural award in the preceding 12 months; (2) department 
administrators that assist faculty with proposals; and (3) central post-award office staff. A total of 
532 individuals were invited to participate, including 463 faculty, 53 department administrators, 
and 16 staff of the central post-award office. This study was not designed with a hypothesis, but 
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Table 1. Office of Research Administration (ORA) Metrics Tracked 


Core Service 

Metrics Tracked 

Proposal Submission 

ORA is first contacted about proposal 

ORA conducts initial review of proposal (if applicable) 

Number of revisions 

Internal forms received by ORA 

Full proposal received by ORA 

Proposal submitted to the agency 

Number of submissions (if applicable) 

Re-budget Requests 

ORA receives re-budget request 

ORA contacts department for revisions (if applicable) 

Department sends revisions (if applicable) 

ORA contacts agency for approval (if applicable) 

Number of contacts with the agency 

Agency returns determination (if applicable) 

ORA routes re-budget to post-award office 

No Cost Extensions 

ORA receives no cost extension request 

ORA contacts department for additional information (if applicable) 
Department sends additional information (if applicable) 

ORA contacts agency for approval (if applicable) 

Number of contacts with agency (if applicable) 

Agency returns determination (if applicable) 

ORA finalizes no cost extension and routes internally 

Subawards (Tufts as 
prime) 

ORA receives notification that sub-award is needed 

ORA contacts department for additional information (if applicable) 

ORA has complete information for sub-award 

ORA sends draft to department for review (if applicable) 

Department returns draft to ORA (if applicable) 

ORA sends sub-award to other party 

ORA follows up with other party (if applicable) 

Number of follow ups with other party 

ORA receives request for changes from other party (if applicable) 

ORA responds to other party (if applicable) 

Number of drafts exchanged (if applicable) 

Sub-award is executed by the other party 

ORA signs sub-award 

Fully executed sub-award distributed internally 


rather to gather information about current levels of satisfaction with ORA that could later be 
used as a benchmark for evaluations and an opportunity for improvement. 

The satisfaction survey (see Appendix A) was sent electronically using the Qualtrics online survey 
tool. It consisted of nine questions grouped as follows: two demographic questions (campus 
location, faculty or staff); three main questions designed to gather information about the volume 
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of grant activity and frequency of the individual’s interactions with O RA; a general question about 
overall satisfaction with ORA services, followed by a detailed question about the respondents’ 
satisfaction; and two open-ended questions asking respondents to explain their other answers. 

The detailed satisfaction question presented a list of ten service metrics and asked respondents 
to rank satisfaction with each using a five-point Likert scale ranging from very satisfied to very 
unsatisfied. The service metrics listed for evaluation were: turnaround time; status updates; 
communication skills; attention to details; knowledge/expertise; professionalism; collegiality; 
responsiveness to customers’ needs; help in managing the proposal time frame well; and quality 
and usefulness of information on ORA website. These metrics were selected for inclusion in the 
survey after conducting a thorough literature review of the performance measures used by other 
research administration offices in their self-evaluations, as well as the analysis of several existing 
survey instruments used by other universities in evaluations of their sponsored programs services 
(Utah State University Survey of the Division of Sponsored Programs; Barry University Office of 
Grants and Sponsored Programs Evaluation Survey). 

The first of the two open-ended questions asked respondents to explain the reasoning behind any 
or all of the qualitative satisfaction scores they had assigned to the service metrics listed in the 
survey. The second gave them an opportunity to express any ideas and opinions about what ORA 
could do to further improve pre-award research administration at Tufts. 

Of 117 participants that clicked on the survey link, 83 people (71%) completed the survey, 
translating to an overall completion rate of 15.6%. Faculty comprised 63% of respondents, while 
staff comprised the remaining 37%. Overall, 52 faculty members and 31 staff completed the 
survey. Of note, the survey received 82 responses to one or more of the open ended questions: 
46 participants (55% of all responses) commented on what impacted their rating(s), and 36 
participants (43% of all responses) commented on what ORA could do to better assist them with 
their pre-award needs. 

Results. The short format required only a brief time and effort investment from the participants, 
potentially increasing the number of responses. While almost half of administrators (45%) who 
were invited to participate in the survey completed it, faculty members participated in smaller 
numbers (11% of those who were invited completed the survey). In analyzing the data, we noted 
that many participating faculty held a stronger opinion, either positive or negative, regarding 
ORA services, while the administrators’ feedback was more evenly spread across the satisfaction 
spectrum. While this survey was not designed to explore in detail the services that were evaluated, 
results showed that it did provide a window for customers to express their satisfaction, both 
quantitatively and qualitatively. 

The survey created a snapshot of faculty and staff satisfaction with ORA on various metrics, 
giving a baseline that can be used in future surveys, especially those following major change. In 
addition to quantitative information, insightful qualitative feedback on ORA services was also 
received. Overall, the majority (67%) of respondents were either very satisfied (28%) or satisfied 
(39%). An additional 24% of respondents were neutral (neither satisfied nor dissatisfied). A total 
of 9% (8 participants) were dissatisfied (7%) or very dissatisfied (2%), with all but one of the 
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dissatisfied respondents being faculty. In service metrics, collegiality and professionalism ranked 
highest in terms of satisfaction. Attention to detail solicited the most responses with some degree 
of dissatisfaction. 



Background. Founded in 2004, the Office of Proposal Development (OPD) at Tufts University 
is a central research development office that offers grant writing and proposal preparation 
assistance to faculty. Solely a pre-award office, services currently offered include grant writing; 
proposal editing and commenting for content; assistance with several other aspects of proposal 
submission, such as project management; funding opportunity searches and database training; 
and a variety of trainings and workshops focusing on aspects of proposal preparation. 

Methods. For the purposes of this survey, faculty who had availed themselves of OPD services 
in the period between 2009 and 2014 and were still employed by Tufts were considered part of 
the target population, totaling 119 faculty. Faculty who met this definition were approached via 
email to ask for their participation. The survey remained open for a period of six weeks, with an 
initial request and three reminders sent via email. Faculty who consented to take the IRB-approved 
survey were directed to an online instrument built using the Qualtrics online survey tool. 

The OPD provides skill-building training, as well as a ‘learn to fish’ model of proposal editing. 
In the ‘learn-to-fish’ model, the grant writer offers comments and edits to successive drafts of 
a proposal, engaging faculty in an iterative process that improves the current proposal while 
explaining and demonstrating best practices. The end goal of this process is to improve the faculty 
member’s proposal writing skills over successive interactions with the office. Given these services, 
constructs gathered from the literature (Campbell & Longo, 2002; Hoyt & Howard, 1978; 
Reif-Lehrer, 1992; Sridhar, 2009; Stein, Clair, Lebeau, Prochaska, Rossi, & Swift, 2012; Wasby, 
2001) led to the inclusion of metrics relating to satisfaction with services provided, as well as to 
what extent faculty felt that their own proposal submission skills and confidence had improved 
after their experience working with the OPD. The survey used a combination of quantitative and 
qualitative questions to assess two major questions associated with these metrics, both having to 
do with the perceived benefits of using OPD programs: (l) To what extent, if at all, were you 
satisfied with the services you received from the OPD during your proposal preparation?; and (2) 
To what extent, if at all, do you perceive that your skills in grant writing and proposal submission 
have improved as a result of your work with the OPD ? 

This mixed-methods satisfaction survey (see Appendix B) asked a total of 23 questions, broken into 
four sections: (l) background information, which asked information about name (not required, 
as it removed anonymity) and rank, time at Tufts, number of grants applied to and received, and 
type of assistance received; (2) experience with the OPD, which asked both quantitative (on a 
five-point Likert scale from very satisfied to very dissatisfied) and qualitative questions relating to 
satisfaction with the OPD in general and with specific services; (3) grant submission post-OPD 
support, which asked quantitative (yes and no, as well as four- or five-point Likert scales tied to 
the construct of each question) and qualitative questions relating to the faculty’s perception of 
their grant application preparation abilities after their experience working with the OPD; and 
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(4), a conclusion section with an open-ended space to provide feedback. The survey was designed 
so that none of the questions required answers to advance to the next question or section. It 
was hypothesized that working with the OPD a) has a positive impact on confidence and skills 
related to pre-award proposal submission; b) increases faculty confidence in the preparation of 
grant applications, and c) is perceived as having a positive impact on faculty’s ability to prepare 
competitive grant proposals. 

Results. A total of 41 faculty clicked on the survey link, with 33 (27.7% of those who were 
invited) completing the full survey. Controlling for responses inputted over multiple days or 
hours, indicating that the survey was not completed in one sitting, the mean response time was 
approximately 10 minutes, with a median response time of approximately 7 minutes. Six responses 
of 33 were removed from analysis for lack of validity. These responses fell into two categories. 
Some provided feedback about other pre- or post-award support offices in place of or in addition 
to the OPD, leading to a lack of clarity about which answers referred to OPD support. Other 
faculty indicated they did not recall that they had received OPD support. The 27 valid responses 
represented 22.7% of the target population. 

Of services provided by the OPD, grant writing/editing assistance was the most commonly 
used service, with 92.6% of respondents reporting that they had received writing/editing help. 
Sixty three percent of respondents report receiving help with extra-narrative elements (such 
as biographical sketches, resources pages, and letters of support), while 37% received project 
management assistance. Overall, 92.6% of respondents were very satisfied (N = 17) or satisfied 
(N = 8). Of the services offered, respondents were most satisfied with editing (96.3% very satisfied 
or satisfied, out of 27) and extra-narrative support (90.5% very satisfied or satisfied, out of 21.) 
Further, 96.3% of respondents would recommend the OPD to a colleague, and 96.3% would also 
use OPD services again. 

Results also showed an increase in confidence in respondents’ ability to prepare competitive 
grant submissions after their experience working with the OPD. Twenty-three of 27 respondents 
reported being very confident (N =17) or confident (N = 6) in their ability to prepare competitive 
grant submissions; 77.8% of respondents reported that in their perception their experience 
working with the OPD had a high (N = 11) or moderate (N = 10) impact on their confidence 
level. An additional 6 respondents reported no impact. Additionally, 85.2% reported that their 
experience working with the OPD had a high (N = 14) or moderate (N = 9) impact on their 
ability to prepare competitive grant proposals. This indicates that the OPD’s intervention has 
a positive impact on proposals that are not prepared using the office, an important finding that 
helps to confirm the ‘learn to fish’ model used by the office does indeed create perceived effects 
on faculty. One third of respondents (N = 9) reported using an OPD tool to teach a colleague, 
which indicates that OPD has been able to assist not only the faculty they reach directly, but also 
those faculty’s colleagues. Thus, the study results provided confirmation of the office’s wider than 
anticipated impact on the pre-award research enterprise at Tufts. 

Results of this study support the hypotheses for the sample population, in that a majority of 
faculty who responded found that working with the OPD increased their confidence in grant 
submission and had a high or moderate impact on their ability to prepare competitive grant 
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proposals. Further, as anticipated, the pairing of long answer qualitative answers with related 
quantitative questions assessing key metrics led to a more detailed set of responses and more 
specific answers to qualitative questions, adding nuance and depth to the data overall. 

Discussion 

Utility and limitations of assessments using qualitative, quantitative, and mixed methods 

The case studies described herein featured assessments using (l) quantitative metrics; (2) a 
quantitative satisfaction survey with limited qualitative questions; and (3) a mixed-methods 
survey that evaluated both satisfaction and learning using quantitative metrics and equally 
weighted qualitative responses. The type of findings and inferences that were made possible by 
the methods used, or the lack thereof, are discussed below. 

Case Study 1: ORA Metrics Study. Results of this quantitative study were found to be rich 
and immediately useful in adjusting operational elements to improve work processes. This 
utility demonstrates the value of using quantitative methods tied to performance metrics to 
assess research administration tasks relating to timing and rounds of review. Such assessments 
can be implemented internally within research administration offices without the involvement 
of customers. They are the least time consuming assessments of the three we evaluated in terms 
of time to conduct (low) and demands on faculty and departmental research administration staff 
(none). FFowever, the sole focus on operational dynamics failed to represent/assess the cultural 
dynamics. In this study, the lack of customer involvement makes it impossible to explore the 
departmental difficulties associated with each of the tracked processes. 

Case Study 2: ORA Customer Satisfaction Survey. Findings from this quantitative satisfaction 
survey with limited qualitative responses demonstrated that involving customers gave access to 
broader data about the customer experience, adding the element missing in an internally focused 
qualitative-metrics-only assessment. Limitations of this method include a lack of actionable 
knowledge if not paired with a quantitative metrics assessment such as that in the ORA Metrics 
Study, which enables more thorough assessment by providing context for customer’s answers. 

Case Study 3: OPD Satisfaction Survey. Results of this mixed-methods survey with 
quantitative metrics and equally weighted qualitative responses demonstrated that when paired 
with quantitative outcomes that detail levels of satisfaction with specific services, the use of 
interspersed qualitative questions added richness and nuance to the analysis of what satisfies 
faculty. Narrative responses allowed faculty to explain why they were satisfied with OPD services. 
Responses such as, “ [ejxtremely professional, courteous, competent and timely in terms of editing 
of research narrative, preparation of biosketches, and providing tactical and strategic advice on 
proposal structure” not only added depth of assessment, but were also useful to determine which 
OPD service provisions provide satisfaction. Limitations of this method include a larger time 
commitment for the OPD staff to review qualitative survey results, as well as the participants 
themselves to complete it. The amount of time necessary from participants, as well as the 
perception that a qualitative survey will be time consuming, may also depress responses. 
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Metrics and competitive advantage 

Metrics such as those tracked in the three described studies have a dual function that is critical to 
the evaluation of an institution’s central and departmental research administration offices, either 
separately or collectively, the combination of which create the research administration enterprise. 
First, they allow the institution to assess its research administration resources from the perspective 
of how they aid in the development of unique core organizational competencies (Rothaermel, 
2013, pp. 88-90). The institution can use this resource-based view to support future investments 
in those resources needed to gain or sustain competitive advantage against peer institutions, or 
justify investments in research administration resources that have previously been made. Second, 
they allow the institution to evaluate alignment between its research administration goals, tasks, 
and people-information to optimize performance. This dual approach equips the institution to 
better position its research administration enterprise for success because it combines an evaluation 
of resources with information on performance. 

Resources are more likely to lead to competitive advantage when they are Valuable, costly to 
Imitate, Rare, and Organized to capture value (the VIRO framework) (Rothaermel, 2013, p. 91). 
Valuable resources correlate with reduced administrative burden and/or increased satisfaction as 
both increase the customers perceived value of the service provided (Rothaermel, 2013, p. 92). 
Such resources can be uncovered using quantitative metrics such as those outlined in the ORA 
Metrics Study. An outcome of that study was to redirect effort from day-to-day activities to re¬ 
mapping some of the processes that were tracked to remove bottlenecks. While this required a 
significant time commitment in an office that was understaffed, it improved service delivery and 
reduced administrative burden, increasing the value gained from ORA’s resources. The value of 
resources can also be discovered through expressed satisfaction such as that qualitatively captured 
using the metrics outlined in the OPD Satisfaction Survey. 

Resources that are rare are those that allow an institution to do better than its peers, or those 
that are used by an institution in a unique way compared to its peers. Costly to imitate resources 
are those that are expensive for peers to replicate in terms of procurement or time needed for 
implementation (Rothaermel, 2013, pp. 92-93). Satisfaction metrics like those outlined in 
the ORA and OPD Satisfaction Surveys can be used to benchmark an institutions resources 
compared to peer institutions. The OPD Satisfaction Survey demonstrated that services offered 
by this office are rare because of the long-term value that faculty received as result of the ‘learn to 
fish’ model. Another key finding, each staff member’s contribution to faculty’s grant development 
experience, highlighted the diverse and specialized skills and training of each staff member. The 
latter also demonstrated that OPD’s resources are costly to imitate given the staff’s specialized 
skill sets and the time investment necessary to cultivate successful relationships with faculty. These 
findings justify prior investments, and provide support for future investments by demonstrating 
that OPD resources are a source of competitive strength for the University. 

In order to capture the most value from resources, there must be organizational structures and 
systems in place that exploit their competitive potential (Rothaermel, 2013, p. 94). Quantitative 
metrics, such as those outlined in the ORA Metrics Study, can be used to understand how structures 
and systems are being used, highlight where processes can be linked and consequentially, how 
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prior investments can be maximized and better utilized. Findings from this study determined that 
the no-cost extension review provides an opportunity for linking with the subaward process as it 
can serve as a prompt for extensions of subcontracts issued under the award. It is also a checkpoint 
for common areas of noncompliance, such as change in scope and personnel effort during the no 
cost extension period. This led to the revision of ORA’s no-cost extension form to include these 
additional data elements. The ORA Metrics Study also identified that more value could be gained 
by utilizing the systems housed in other central offices at Tufts in the re-budget request process. 
Re-budget requests are now routed to the post-award office first, and to ORA second and only in 
instances where agency approval is necessary. This has exploited the competitive potential of these 
resources while maximizing prior investments. 

For a university research administration office, metrics such as those outlined in the ORA Metrics 
Study and Customer Satisfaction Survey can be used to determine where there is misalignment 
between tasks, goals, and/or people. According to the Organizational Alignment Model (Figure 
1), organizations develop an overarching strategy and define their goals in response to the external 
environment. To accomplish these goals, they create tasks and hire people to perform these tasks. In 
order for the strategy to be successful there must be alignment between the defined goals, the created 
tasks, and the hired people. It is from this alignment that the organization gains competitive strength 
(Lane, Maznevski, DiStefano, & Dietz, 2009, pp. 179-185). Ideally, identified misalignments 
will lead to changes that will help build the skills needed to perform the tasks required to meet 
an organizations goal(s), creating the alignment that is critical in driving future performance. 
An example of this type of alignment, the ORA Metrics Study highlighted that a majority of the 
bottlenecks in the existing processes for re-budget and no-cost extensions requests were attributable 
to a disconnect at the department level. More specifically, the disconnect was present between the 
associated tasks and the skill-set of the individuals performing these tasks. As an outcome of this 
study, as well as other assessment initiatives, efforts are currently underway to create a standard 
level of performance. This will be accomplished by re-defining and re-designing positions so that 
they have a stronger research administration 
orientation and by providing more formalized 
and institution-driven training. 

Qualitative feedbackis also useful in discovering 
misalignment between faculty’s perception of 
the goals established by an organization for 
its research administration operation, and the 
actual goals, tasks, and people assigned to the 
research administration enterprise. This issue is 
typified by the results of the OPD Satisfaction 
Survey, as faculty respondents requested 
services that did not align with university 
or office-set goals. Knowledge about this 
disconnect allows for better communication 
of existing university and office goals and for 
evaluation of the office tasks for alignment Figure 1. Organizational Alignment Model. 



Note. Lane et al., 2009 
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with faculty’s goals. Greater alignment between the two can alter the respective contribution of 
OPD and faculty to the task at hand, and ultimately drive performance. 

Conclusion 

These findings demonstrate that the types and combinations of methods most useful in evaluating 
an office of research administration are dependent on several factors, including the goal of 
evaluation, types of available baseline data, the amount of time and institutional support available 
to the investigator, and the amount of time and effort that can be asked of survey respondents. 
Used in combination based on the needs of a given office, the approaches employed in each case 
study offer examples of how to usefully assess the impact, strengths, and weaknesses of a research 
administration office, and can be used to gain or sustain competitive advantage and position 
the research administration enterprise for future success. While it is recognized that evaluation 
metrics can be influenced by external forces (i.e. the PESTEL framework), the methods used in the 
three described case studies were not designed to assess the potential impact of said f actors. These 
forces were instead referenced to highlight the need to look internally at the institution’s resources 
when evaluating its research administration enterprise. Additional research on external factors is 
warranted to explore ways to assess the impact of such factors on research administration metrics. 
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Appendix B 

Tufts Office of Proposal Development Satisfaction Survey 

Thank you for agreeing to participate in this survey! The Tufts Office of Proposal Development 
(OPD) is evaluating the satisfaction of faculty with the OPD and the services that we provide, as 
well as the perceived effects of our services. The survey should take between 15-20 minutes, and 
will include questions about your background in grant submission, your experience working with 
the Tufts OPD, and any new grant submissions since that interaction. 

Background Information 

The questions in this section will ask you for information about yourself and your position at the 
university. You will also be asked questions about your previous grant submission experience, and 
the types of assistance you have received from the Tufts OPD. 

1. We will start with a few questions about your position at Tufts. If you would like anonymity, 
please skip these questions. If you choose to answer this question you are providing consent for 
the investigator to contact you with follow up questions based on your responses below, regarding 
clarifications of or expansions on your answers. 

• Name 

• Position 

• Date of Hire 

2. Please select the number of grant applications on which you were listed as Principal Investigator 
(PI) (or equivalent) either at Tufts or at another institution, during the period between 2003 
and 2014. 

. 0-1 
.2-3 
.4-6 
.7-9 
. 10 + 

3. Of the grant applications above, approximately how many have been funded? 

. 0-1 

. 2-3 
. 4-6 
. 7-9 
. 10 + 
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4. Please check all types of assistance you received from the OPD during your submission. If you 
received a type of assistance that is not listed, please write it in below. 

• Funding Search/Database Instruction 

• Commenting/Editing of Research Plan 

• Project Management 

• Extra-Narrative (e.g. Biosketch, Resources) 

• Seminar 

• Other 

Experience with the Office of Proposal Development 

In this section you will be asked to discuss your experience working with the OPD. Some questions 
will ask you to rate your satisfaction with experiences and services on a scale. Others will ask you 
to describe your thoughts on the services provided. 

la. Please rate your satisfaction with the overall services provided to you by the OPD, using the 
scale below. 

• Very Satisfied 

• Satisfied 

• Neutral 

• Dissatisfied 

• Very Dissatisfied 

lb. Please explain your rating in the space below, citing specific examples where available. 

2a. Please rate your satisfaction with the specific services listed below, with Very Satisfied 
indicating that the service provided was excellent, and Very Dissatisfied indicating that the service 
provided was poor. If you did not receive a service, please select ‘Not Applicable.”(Note, scale for 
all Very Satisfied, Satisfied, Neutral, Dissatisfied, Very Dissatisfied) 

• Funding Search 

• Project Management 

• Readings/Commenting 

• Extra-Narrative (e.g. Biosketch) 

• Seminar 

2b. Please explain your ratings in the space below. What exchanges with members of the OPD team 
were either particularly satisfying or dissatisfying ? Please provide specific examples if available. 

3a. Please think back to the submission(s) with which you were assisted by the OPD. What 
elements of support would you describe as the most helpful to you during the proposal 
development process ? 
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3b. What elements of OPD support would you describe as the least helpful in the completion of 
your proposal? 

Grant Submission Post-OPD Support 

This final section of the survey will concern your perception of your grant application preparation 
abilities after your experience working with the OPD. Some questions will ask you to rate your 
abilities on a scale; others will ask you to describe your thoughts. 

1. Thinking about your experience working with the OPD, what changes, if any, do you perceive 
in your ability to prepare grant submissions? 

2a. Please rate your current confidence level in your ability to prepare competitive grant 
submissions, using the scale below. 

• Very Confident 

• Confident 

• Neutral 

• Unconfident 

• Very Unconfident 

2b. Please rate your perception of how working with the OPD has affected your confidence level 
in your ability to prepare competitive grant submissions, if at all, using the scale below. 

• High Impact 

• Moderate Impact 

• No Impact 

• Moderate Negative Impact 

• High Negative Impact 

3. Please rate your likelihood of applying for future grant funding, using the scale below. 

• Very Likely 

• Likely 

• Unlikely 

• Very Unlikely 

4. Please rate your perception of how your work with the OPD has affected your likelihood of 
applying for future grant funding, if at all, using the scale below. 

• High Impact 

• Moderate Impact 

• No Impact 

• Moderate Negative Impact 

• High Negative Impact 
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4b. Please rate the impact that working with the OPD has had on your ability to prepare 
competitive grant submissions, if any. 

• High Impact 

• Moderate Impact 

• No Impact 

• Moderate Negative Impact 

• High Negative Impac 

5. Have you used a tool or strategy learned from the OPD to assist or instruct a colleague? 

. Yes 

. No 

5b. If yes, please discuss below, using specific examples if possible. 

6. Please rate your likelihood of suggesting that a colleague contact the OPD for support, using 
the scale below. 

• Very Likely 

• Likely 

• Unlikely 

• Very Unlikely 

7. Please rate your likelihood of using OPD services again, using the scale below. 

• Very Likely 

• Likely 

• Unlikely 

• Very Unlikely 

Conclusion 

Please use this space to provide any feedback on the survey instrument, or other comments about 
your experience with the Office of Proposal Development. 

Do you consent to the inclusion of text from your responses in reports, papers, or presentations? 
Any attributions made will remain anonymous. 

. Yes 
. No 

This concludes the Tufts Office of Proposal Development New Faculty Satisfaction Survey. 
Thank you very much for your time and effort in answering our questions! 
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